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Abstract 

This  study  investigates  the  relationship  between  risk 
alignment  and  cost  performance  in  DoD  procurement.  Both  the 
government  and  the  contractor  conduct  independent 
assessments  of  the  perceived  risk  in  a  contract.  The 
results  of  inaccurate  risk  assessments  are  undesirable 
consequences,  one  of  which  is  the  inability  to  achieve  cost 
targets.  If  both  the  contractor's  and  the  government's 
perceptions  of  risk  are  in  alignment  with  the  actual  risk 
involved  in  a  contract,  then  we  would  expect  minimal  impact. 
If,  however,  either  or  both  parties  fail  to  accurately 
assess  the  actual  risk,  then  we  would  expect  consequences, 
such  as  cost  overrun/underrun.  The  level  of  cost 
overrun/underrun  can  be  determined  by  examining  the  level  of 
alignment  between  the  actual  risk  and  each  party's 
perception  of  the  risk. 

Results  indicate  a  significant  potential  for  cost 
growth  when  both  the  government  and  the  contractor 
underestimate  the  actual  risk.  In  this  context,  the  level 
of  cost  growth  experienced  on  a  contract  is  a  reflection  of 
the  adequacy  of  the  parties'  risk  assessments.  Failure  to 
accurately  assess  risk  could  result  in  a  lack  of  sufficient 
"safeguards"  necessary  to  help  mitigate  the  damaging  effects 
of  risk  inherent  in  a  contract. 
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AN  ANALYSIS  OF  THE  RELATIONSHIP  BETWEEN 


RISK  ALIGNMENT  AND  COST  PERFORMANCE 


I .  Introduction 


GENERAL  PURPOSE 

The  concept  of  risk  pervades  every  action  of  daily 
life.  Life  requires  choices  and  choices  involve  risks. 

Every  day  we  subject  ourselves  to  the  risk  of  death  by 
getting  in  our  cars  and  driving  down  the  road.  While  we 
cannot  avoid  risks  completely,  we  can  minimize  the  risks  we 
face . 

Although  everyone  makes  daily  decisions  involving 
risks,  not  everyone  makes  daily  decisions  resulting  in  risks 
associated  with  thousands  of  dollars  or  thousands  of  lives . 
Political  leaders  and  managers  of  large  business  firms, 
however,  are  faced  with  such  decisions  with  regularity.  In 
addition,  they  are  ultimately  responsible  for  the 
consequences  of  the  decisions  they  make.  How  important, 
then,  is  it  for  organizational  leaders  to  understand  the 
elements  of  risk  associated  with  their  decisions?  I  would 
consider  it  critical.  To  ignore  potential  risks  and  make 
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decisions  with  a  laissez-faire  attitude  is  not  management; 


it  is  gambling.  Since  risk  management  recognizes  the 
potential  impact  (damage)  of  risky  decisions,  it  can  take 
proactive  actions  (safeguards)  to  reduce  risk. 

For  business  and  government  leaders,  the  effect  of  not 
recognizing  and  planning  for  risk  is  to  subject  themselves 
to  uncontrollable  consequences.  Risk  is  associated  with 
future  events  and,  as  such,  contains  elements  beyond  our 
control.  The  proper  use  of  risk  analysis  and  risk 
management  techniques,  however,  can  work  to  mitigate  those 
risks  to  the  point  where  the  outcomes  are  within  acceptable 
limits . 

The  underlying  premise  is  that  decisions  involve  risks. 
We  can  choose  to  either  ignore  those  risks  and  accept  the 
consequences,  or  we  can  take  proactive  steps  toward 
understanding  the  risks  associated  with  our  decisions  and 
developing  ways  to  mitigate  those  risks,  thereby  maintaining 
acceptable  results. 

One  of  the  most  damaging  results  of  poor  risk 
management  within  the  DoD  is  the  issue  of  cost  growth.  The 
average  cost  overrun  on  defense  contracts  has  been  estimated 
at  40  percent  (Gansler,  1989:  4).  Considering  the  enormity 
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of  acquisition  expenditures  within  the  DoD,  this  estimate 


could  result  in  dollar  overruns  of  $37B  per  year. 

Extensive  research  has  been  conducted  to  determine  the 
cause  of  cost  growth  in  Federal  acquisition  programs 
(Drezner,  1995; Fletcher  &  Young,  1994 ; Singleton,  1991).  No 
one,  however,  has  developed  a  single  definitive  explanation 
for  cost  overruns.  A  1993  Rand  study  set  out  to  identify 
factors  affecting  cost  growth  (Drezner,  et.al,  1995:  xi)  . 
They  examined  aspects  ranging  from  program  length,  size,  and 
maturity  to  management,  budget,  and  political  decisions. 
Their  overall  observation:  "the  problem  of  cost  growth  does 
not  have  a  'silver  bullet'  policy  response"  (Drezner,  et.al, 
1995:  xiii) .  In  other  words,  many  factors  affect  the 
problem  of  cost  overruns,  yet  no  one  factor  can  individually 
be  labeled  as  the  defining,  overarching  precedent  to  cost 
growth . 

Although  much  research  has  been  accomplished  to. 
investigate  the  causes  of  cost  growth,  little  has  been 
conducted  to  address  how  poor  risk  assessment  affects  cost 
growth.  The  intent  of  this  study  is  to  investigate  the 
effect  of  inaccurate  risk  assessment  on  the  government's 
ability  to  achieve  cost  targets. 
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BACKGROUND 


Each  party  to  the  contract,  the  government  and  the 
contractor,  conducts  an  independent  assessment  of  the  risks 
associated  with  a  contract.  This  risk  assessment  is  a 
reflection  of  their  perception  of  the  level  of  risk  faced. 
Poor  risk  assessments  can  lead  to  consequences,  most  of 
which  are  damaging  in  nature.  Poor  risk  assessments  can 
also  lead  to  consequences  which,  on  the  surface,  appear  to 
be  beneficial.  For  instance,  an  assessment  of  risk  which  is 
higher  than  the  actual  risk  could  result  in  an  overall  cost 
underrun.  While  this  initially  appears  to  be  a  desirable 
outcome,  it  in  fact  means  funds  were  under-utilized, 
possibly  to  the  detriment  of  other  programs. 

One  way  to  investigate  the  impact  of  risk  assessment  on 
cost  performance  is  by  analyzing  the  alignment  between  each 
party' s  (government  and  contractor)  perception  of  risk  and 
the  actual  risk  involved  in  a  contract.  If  both  the 
contractor's  and  the  government's  perceptions  of  risk  are  in 
alignment  with  the  actual  risk  involved  in  a  contract,  then 
we  would  expect  minimal  impact.  If,  however,  either  or  both 
parties  fail  to  accurately  assess  the  actual  risk,  then  we 
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would  expect  consequences,  such  as  cost  overrun/underrun. 
Consequently,  it  may  be  argued  that  cost  overrun/underrun 
is,  in  part,  a  function  of  the  alignment  between  the  actual 
risk  and  each  party's  perceptions  of  the  risk.  In  instances 
where  risk  is  underestimated,  the  expectation  is  that  the 
contract  will  experience  a  cost  overrun,  because  the  parties 
have  failed  to  adequately  provide  "safeguards"  to  help 
minimize  the  consequences  of  the  actual  risk  level. 
Conversely,  if  risk  is  overestimated,  the  expectation  is 
that  managers  have  under-utilized  their  funds  and  the 
contract  will  experience  a  cost  underrun. 


RESEARCH  DESIGN 

The  Defense  Acquisition  Executive  Summary  (DAES) 
database  contains  historical  information  on  completed  and 
ongoing  Department  of  Defense  (DoD)  contracts.  The  data  is 
summarized  in  the  form  of  Cost  Performance  Reports  (CPRs) 
and  submitted  to  the  Office  of  the  Under  Secretary  of 
Defense  for  Acquisition  &  Technology  (OUSD(A&T))  on  a 
quarterly  basis  (DoD,  1991) .  The  database  contains 
contracts  from  all  three  military  services  and  covers  a  wide 
variety  of  contract  types  and  end-items. 
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The  dependent  variable  of  interest  for  this  study  is 
the  final  cost  variance  of  contracts  in  the  DAES  database. 

At  contract  completion,  the  overrun  or  underrun  is  the 
difference  between  the  total  budget  for  all  the  work  on 
contract,  termed  the  "Budget  at  Completion"  (BAC) ,  and  final 
actual  cost  of  the  work  performed  (ACWP)  (Christensen,  1993: 
44)  . 

The  independent  variables  of  interest  are  the  measures 
of  each  party's  perceptions  of  risk  and  the  actual  risk 
involved  in  the  contract.  The  measure  of  actual  risk  chosen 
for  this  study  was  the  phase  of  procurement  reported  in  the 
DAES  database.  Contracts  pertaining  to  the  R&D  phase  of 
development  were  coded  as  high  risk,  while  contracts  used 
during  production  were  coded  as  low  risk.  The  reported 
contract  type  was  used  as  the  government's  perception  of 
risk.  Fixed-price  type  contracts  were  used  to  indicate  a 
low  level  of  perceived  risk,  while  cost-reimbursable  type 
contracts  reflected  a  high  level  of  risk  perception. 

Finally,  the  contractor's  perception  of  risk  was  determined 
by  the  initial  percentage  of  management  reserve  (MR)  budget 
reported  in  the  DAES  database.  High  risk  was  viewed  as  an 
initial  MR  >  7.5%,  while  MR  <  7.5%  was  considered  low  risk. 
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The  individual  perceptions  of  risk  and  the  actual  risk 


were  then  organized  into  alignment  combinations.  Two 
»  combinations  were  investigated  in  this  study.  The  first  was 

the  scenario  in  which  both  the  government  and  the  contractor 
overestimated  the  actual  risk.  It  was  believed  that  this 
combination  would  result  in  the  highest  degree  of  cost 
underrun.  The  second  combination  was  the  scenario  in  which 
both  the  government  and  the  contractor  underestimated  the 
actual  risk,  thereby  resulting  in  the  highest  degree  of  cost 
overrun . 

The  procedure  used  to  analyze  the  difference  in  cost 
performance  for  the  two  combinations  was  the  Analysis  of 
Variance  (ANOVA)  technique.  The  ANOVA  was  used  to  compute 
the  mean  overrun/underrun  values  for  each  combination  and 
then  compare  the  means  for  statistically  significant 
differences  at  the  p-value=.05  level. 

FINDINGS 

Analysis  of  the  statistical  results  indicated  that  the 
alignment  of  risk  had  a  significant,  but  inconclusive  effect 

r 

on  the  contract  cost  performance.  In  the  instance  where 
both  the  government  and  the  contractor  underestimated  the 


7 


risk,  the  mean  cost  performance  was  a  22%  overrun. 


Conversely,  an  overestimation  of  risk  by  both  parties 

resulted  an  mean  underrun  of  approximately  1.4%.  * 

The  findings  indicate  that  inaccurate  alignments  of 
risk  can  lead  to  undesirable  cost  performance.  The 
implications  for  DoD  contract  managers  is  that  proper  risk 
management  is  a  critical  element  in  the  effort  to  achieve 
cost  targets.  One  part  of  proper  risk  management  is  the 
ability  to  accurately  analyze  the  actual  risk  in  a  contract 
and  then  take  the  necessary  steps  to  develop  "safeguards" 
capable  of  mitigating  the  damaging  consequences  of  that 
inherent  risk.  Failure  to  adequately  align  the  perceptions 
of  risk  with  the  actual  risk  has  been  shown  by  this  study  to 
create  a  tendency  towards  poor  cost  performance. 


ORGANIZATION  OF  THE  STUDY 

Chapter  II  establishes  the  theoretical  foundation  for 
the  study.  The  general  concept  of  risk  and  risk  management 
from  the  government  and  the  contractor  perspective  is 
discussed.  The  chapter  concludes  with  the  development  of 
two  hypotheses  used  to  test  the  relationship  between  risk 
alignment  and  cost  performance.  Chapter  III  details  the 
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methodology  and  research  design  of  the  study.  The  validity 
and  measurement  techniques  for  the  dependent  and  independent 
variables  are  established.  The  analytical  model  used  to 
test  the  hypotheses  is  presented.  Chapter  IV  reports  the 
results  of  the  statistical  analysis.  The  study  concludes 
with  a  discussion  of  the  implications  of  the  results  and 
recommendations  for  future  studies  in  Chapter  V. 


II.  Literature  Review 


OVERVIEW 

Each  party  to  a  contract,  the  contractor  and  the 
government,  conducts  an  independent  assessment  of  the  risks 
involved  in  a  program.  The  accuracy  of  those  assessments  is 
reflected  in  the  resulting  consequences,  one  of  which  is 
cost  performance.  The  degree  to  which  either  or  both 
parties  calculates  risk  accurately  can  be  seen  by  examining 
the  level  of  cost  underrun/overrun  experienced  on  a 
particular  contract. 

This  chapter  will  present  the  concept  of  risk  as  it 
applies  to  Department  of  Defense  (DoD)  acquisition. 

Following  a  general  discussion  of  risk,  descriptions  of  the 
government's  and  contractor's  risk  management  processes  are 
provided.  The  chapter  concludes  with  two  hypotheses 
established  to  investigate  the  link  between  risk  alignment 
and  cost  growth. 


THE  GENERAL  CONCEPT  OF  RISK 

The  purpose  of  this  section  is  to  investigate  the  issue 
of  risk  and  determine  the  role  risk  management  plays  in  the 
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formulation  of  business  strategy.  This  section  begins  with 


conceptual  definitions  of  risk  and  then  follows  with  a 
discussion  of  risk  management  and  risk  analysis.  The 
section  concludes  with  two  examples  of  the  business-related 
outcomes  associated  with  various  forms  of  risk. 

Definitions  of  Risk 

Webster  defines  risk  as  exposure  to  possible  loss  or 
injury.  This  definition  of  risk  contains  three  key 
elements-exposure,  possibility,  and  loss.  First,  for  an 
event  or  decision  to  involve  risk,  there  must  be  some 
potential  loss  to  the  decision  maker.  Second,  the  element 
of  possibility  requires  that  there  be  some  form  of  "chance" 
associated  with  the  outcome.  A  situation  involving  complete 
certainty  of  loss  is  not  a  risk.  Finally,  the  definition 
refers  to  the  concept  of  exposure.  The  exposure  element  of 
the  definition  reflects  the  notion  that  the  decision  maker 
has  the  ability  to  take  actions  which  may  increase  or 
decrease  the  severity  of  the  loss  (MacCrimmon  &  Wehrung, 
1986:  9). 

Thus,  according  to  this  definition  of  risk,  there  are 
three  components — the  degree  of  loss,  the  chance  a  loss  will 
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occur,  and  the  willingness  or  decision  to  expose  oneself  to 
loss.  Efforts  to  diminish  the  risk  of  any  decision  or 
behavior  must  then  focus  on  reducing  one  of  these  elements. 
The  degree  of  risk  associated  with  an  action  can  be  thought 
of  as  being  directly  proportional  to  the  magnitude  of  the 
loss,  the  probability  (or  chance)  loss  will  occur,  and  the 
amount  of  exposure  to  loss  (MacCrimmon  &  Wehrung,  1986:  10) . 

The  preceding  definition  of  risk  focuses  primarily  on 
the  negative  impacts  of  "risky"  decisions.  In  his  book.  The 
Business  of  Risk,  Peter  Moore  reminds  us  that  risk  also 
involves  the  hope  of  some  benefit  or  gain  (Moore,  1986:  3). 
Moore  points  to  a  quote  from  Shakespeare's  Merchant  of 
Venice:  "Men  that  hazard  all  do  it  in  hope  of  fair 

advantages."  Using  hazard  synonymously  with  risk,  Moore 
points  out  that  engaging  in  risky  ventures  involves  the 
potential  for  favorable  outcomes.  Indeed,  from  the  business 
perspective,  this  would  seem  intuitive.  If  taking  risks 
involved  no  potential  for  gain,  then  sound  business  judgment 
would  require  us  to  avoid  risky  endeavors  in  favor  of  the 
status  quo. 
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Conceptually,  risk  can  be  defined  as  a  function  of 


uncertainty  and  damage,  as  shown  below  (Kerzner,  1995:  879) . 

*  Risk  =  /(uncertainty,  damage) 

*  In  order  to  understand  this  definition,  a  distinction  must 
be  made  between  risk  and  uncertainty.  Risk,  in  this 
instance,  refers  to  an  outcome  which  is  subject  to  an 
uncontrollable  random  event  derived  from  a  known  probability 
distribution.  Conversely,  uncertainty  involves  outcomes  in 
which  the  probability  distribution  is  unknown.  For  example, 
the  toss  of  a  die  yields  the  probability  that  each  side  will 
turn  up  one-sixth  of  the  time.  Each  surface  has  an  equal 
chance  of  occurring,  so  the  laws  of  nature,  over  time, 
dictate  a  known  likelihood  of  any  particular  side  landing 
face-up.  Consider  now  a  loaded  die,  the  precise  weighting 
of  which  was  unknown.  Without  conducting  experiments,  we 
would  be  uncertain  of  the  exact  probability  of  any 
particular  face  landing  upturned  (L'Heureux  &  Grant,  1996: 
18-3) .  Consequently,  a  gambler  would  be  taking  a  calculated 

,  risk  when  competing  with  a  fair  die.  Any  bets  placed  on 

loaded  die,  however,  would  result  in  outcomes  which  were 
uncertain. 
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similar  to  Webster's  definition  of  risk,  the  conceptual 
definition  presented  above  involves  some  degree  of  damage 
associated  with  the  outcome.  For  example,  two  projects. 
Project  A  and  Project  B,  may  have  identical  probabilities  of 
failure.  However,  if  the  consequences  of  not  completing 
Project  A  are  greater  than  those  for  not  completing  Project 
B,  then  Project  A  would  be  considered  the  more  "risky" 
situation.  In  general,  as  either  uncertainty  or  damage 
increases,  so  does  risk.  Furthermore,  since  risk  actually 
constitutes  a  lack  of  knowledge  concerning  future  events, 
risk  analysis  must  account  for  the  cumulative  effect  of  the 
outcomes  associated  with  the  risk-taker's  decisions. 

Another  element  of  the  conceptual  definition  is  the 
cause  of  risk.  The  source  of  danger  which  induces  risky 
situations  is  referred  to  by  Kerzner  as  the  hazard  (Kerzner, 
1995:  879).  This  leads  to  the  second  conceptual  equation: 

Risk  =  /(hazard,  safeguard) . 

The  safeguard  refers  to  preventative  actions  which  may  be 
taken  to  overcome  the  hazard.  An  example  is  a  large  pothole 
in  the  road.  The  pothole  is  the  hazard  which  creates  the 
risk  of  damaging  a  vehicle.  A  safeguard  to  this  risk  is 
illustrated  by  the  driver  who  is  familiar  with  the  road. 
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The  familiar  driver  would  slow  down  and  go  around  the  hole, 
in  order  to  reduce  the  risk  of  damage.  As  we  can  see  then, 

*  hazards  are  directly  related  to  risks,  while  safeguards  have 

^  an  inverse  relationship  with  risk.  That  is,  increased 

hazards  will  increase  risk  while  increases  in  safeguards 
(preventative  actions)  tend  to  reduce  overall  risk. 

In  the  business  context,  the  most  direct  way  to 
illustrate  the  concept  of  risk  is  through  the  use  of 
decision  tree  analysis.  Consider,  for  example,  a  company 
considering  the  production  of  a  new  line  of  merchandise. 

The  decision  tree  might  look  like  the  one  shown  in 
Figure  2-1. 


Figure  2-1:  Decision  Tree 
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Inspection  of  the  decision  tree  shows  there  are  two 


types  of  nodes  on  the  tree.  The  square  decision  node 
represents  a  point  in  time  where  a  choice  must  be  made 
between  alternative  courses  of  action.  In  this  instance, 
the  company  may  decided  to  either  build  a  large  plant,  at  a 
cost  of  $200,000,  or  build  a  smaller  plant,  at  a  cost  of 
$100,000.  The  circular  nodes  represent  states  of  nature, 
where  the  outcomes  are  not  entirely  within  the  control  of 
the  company. 

Through  market  research,  the  company  may  have 
determined  that  each  possible  outcome  (demand  rate)  has  a 
certain  respective  probability  of  occurrence.  In  this 
example,  there  is  a  0.5  probability  of  high  demand,  a  0.3 
probability  of  medium  demand,  and  a  0.2  probability  of  low 
demand,  regardless  of  which  plant  is  built.  The  monetary 
values  listed  to  the  far  right  of  the  decision  tree  are  the 
net  values  of  sales  over  the  life  of  the  plant,  relative  to 
each  demand  rate.  They  are  computed  based  on  net  sales 
minus  capital  expenditures,  where  net  sales  for  each  demand 
rate  are  given  as  $800K  for  high  demand,  $400K  for  medium 
demand,  and  $150K  for  low  demand.  In  decision  tree' 
analysis,  they  are  often  referred  to  as  the  payoff  for  the 
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given  combination  of  decisions  and  states  of  nature.  For 
example,  the  payoff  for  the  large  plant,  high  demand 

*  combination  is: 

t  Net  Sales  -  Capital  Expenditures 

=  $800K  -  $200K  =  $600K. 

Notice  for  the  small  plant  decision,  the  payoff  for  the  high 
and  medium  demand  rates  are  identical,  because  the  small 
plant  cannot  produce  enough  to  meet  the  higher  demand  and 
sales  are  therefore  limited  to  the  medium  demand  rate. 

Now,  in  order  to  make  a  business  decision  on  which 
plant  to  build,  the  expected  value  of  each  decision  must  be 
calculated  by  summing  the  products  of  payoff  X  probability 
for  each  of  the  two  possible  decisions.  For  the  large  plant 
decision,  the  expected  value  (EV)  is  given  as: 

EVl  =  ($600K)(.5)  +  ($200K)(.3)  +  ($-50K)(.2)  =  $350K. 
Likewise,  for  the  small  plant  decision: 

EVs  =  {$300K){.5)  +  ($300K){.3)  +  ($50K)(.2)  =  $250K. 

We  can  see,  based  on  the  predictions  given  in  this  scenario, 
the  decision  to  build  the  larger  plant  would  result  in  a 
higher  expected  value  over  the  life  of  the  plant. 

♦ 

Therefore,  the  optimal  business  strategy  in  this  example 
would  be  to  build  the  large  plant. 
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The  preceding  example  illustrates  several  of  the 


elements  of  our  conceptual  definitions  of  risk.  First, 
there  is  uncertainty  involved  in  the  estimates  of  demand 
rates  and  in  the  estimated  net  sales  over  the  life  of  the 
plants.  On  a  more  fundamental  level,  a  real-world  problem 
would  involve  uncertainty  in  a  myriad  of  elements,  such  as 
production  costs,  salaries,  changes  in  technology,  etc. 
Second,  there  is  a  potential  for  damage  associated  with  the 
decision.  There  is  a  20%  probability  that,  having  chosen  to 
build  the  large  plant,  the  company  would  lose  $50K  over  the 
life  of  the  project.  Third,  there  exist  hazards  which  may 
affect  the  risk  of  the  decisions.  Another  company  may 
decide  to  market  a  similar  consumable,  thereby  reducing  the 
anticipated  demand.  Finally,  there  are  potential  safeguards 
the  company  may  use  to  lower  the  risk  associated  with  the 
hazards,  such  as  aggressive  advertising  and  early  entry  into 
the  market . 

Risk  Management 

According  to  MacCrimmon  &  Wehrung,  all  risky  situations 
inherently  contain  three  identifiable  determinants:  lack  of 
control,  lack  of  information,  and  lack  of  time  (MacCrimmon  & 
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Wehrung,  1986:  14) .  If  we  had  complete  control  of  the 
situation,  we  could  always  determine  the  best  possible 
outcome,  and  there  would  be  no  risk.  Similarly,  if  we  had 
perfect  information,  there  would  be  complete  certainty  in 
all  of  our  actions.  Again,  risk  would  not  exist  because  of 
a  lack  of  the  uncertainty  associated  with  unknown 
probabilities.  If  we  had  no  time  constraints,  we  could  wait 
until  the  outcome  of  any  uncertain  event  was  resolved. 

Proper  risk  management,  then,  involves  understanding  these 
three  determinants  and  seeking  ways  to  reduce  their 
undesirable  effects  on  outcomes. 

From  the  business  perspective,  risk  management  is  an 
organized  means  of  identifying  and  measuring  risk  and 
developing,  selecting,  and  managing  options  for  handling 
these  risks.  It  involves  understanding  the  danger  signals 
that  may  indicate  a  project  is  off  track  and  prioritizing 
corrective  actions  as  necessary.  Risk  management  is  not  a 
separate  office  activity,  but  rather  is  one  aspect  of  sound 
technical  management.  Proper  risk  management  implies 
control  of  possible  future  events,  and  is  proactive  rather 
than  reactive.  Proper  risk  management  will  reduce  not  only 
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the  likelihood  of  an  unfavorable  event  occurring,  but  also 
the  magnitude  of  its  impact  (Kerzner,  1995:  880) . 

The  recurring  premise  is  that  the  best  way  to  handle  ' 

risky  situations  is  to  manage  them.  By  this,  I  mean  taking  » 

an  active  stance  in  identifying  potential  problem  areas  and 
then  preparing  a  necessary  course  of  action  to  alleviate  the 
damage  associated  with  risky  decisions.  This  leads  to  the 
topic  of  the  next  section — risk  analysis.  In  order  to 
effectively  manage  risk,  we  must  first  ascertain  what  risks 
exist  and  what  tools  are  available  to  alleviate  risk. 

Risk  Analysis 

The  ultimate  purpose  of  risk  analysis  is  to  identify 
and  mitigate  the  inherent  risks  to  the  point  where  they  are 
manageable.  Accomplishing  risk  mitigation  requires  an 
understanding  of  the  risks  involved  and  the  impact  they  may 
have  on  meeting  desired  objectives.  Kerzner  describes  the 
impact  of  risk  with  the  following  mathematical  illustration 
(Kerzner,  1995:  890): 

Impact  of  Risks (likelihood  of  risk) x (consequence  of  risk) . 

This  relates  directly  back  to  the  conceptual  definition  of 
risk.  Earlier,  we  had  defined  risk  to  be  a  function  of  both 
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the  uncertainty  of  the  situation  and  the  potential  damage. 
The  illustration  presented  here  shows  the  compounding  effect 

♦  these  two  elements  have  on  the  overall  impact  of  risk. 
Reducing  the  impact  through  risk  analysis  would  then  involve 
modeling  the  likelihood  of  event  occurring  and  understanding 
the  consequences  of  that  event  occurring. 

Many  tools  have  been  developed  to  assist  in  the 
analytical  process  of  modeling  risk.  Some  of  them  are 
described  below  (DSMC,  1989:  5-28  -  5-58) . 

Network  Charts  -  a  graphical  display  representing 
the  individual  activities  necessary  for  the  completion 
of  a  project,  the  time  required  for  each  activity,  and 
the  inter-relationships  between  activities. 

Life-cycle  cost  analysis  -  Computerized  models 
used  to  predict  the  effect  a  change  will  have  over  the 
full  spectrum  of  development,  production,  operations, 
and  support  costs. 

Probability  analysis  -  Statistical  modeling  of  the 
uncertainty  associated  with  a  decision  or  event. 

♦ 

Decision  analysis  -  Probabilistic  model  of  the 
expected  monetary  value  of  a  decision. 
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Work  Breakdown  Structure  (WBS)  -  A  process  which 


develops  individual  probability  distributions  for  each 
"work  package"  and  then  aggregates  the  individual  ' 

estimates  into  one  total  program  cost  risk  » 

distribution . 

Consequences  of  Poor  Risk  Management:  2  DoD  Examples 

In  the  early  1980' s  the  U.S.  Army  was  developing  the 
Sergeant  York  air  defense  gun,  formerly  known  as  the 
Division  Air  Defense  gun,  to  fill  a  perceived  void  in  the 
air  defense  of  forward  battle  areas.  Sergeant  York's 
primary  mission  was  to  engage  enemy  helicopters  and  fixed- 
wing  aircraft  involved  in  close-air-support  activities.  It 
was  to  have  a  three  member  crew  and  an  armament  of  twin  40- 
mm.  radar-directed  computer  controlled  guns,  and  a  7.62-mm. 
machine  gun.  The  Army  had  a  planned  procurement  of  618 
production  units  at  a  total  acquisition  cost  of  $4.2  billion 
(GAO,  1983:  1). 

On  August  27,  1985,  the  Secretary  of  Defense  announced  ^ 

he  was  canceling  the  Sergeant  York  program.  Out  of  the  618 
planned  production  units,  the  contractor.  Ford  Aerospace, 
had  delivered  only  64  systems  at  the  time  of  the  Secretary's 
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decision  (GAO,  1986:  1).  The  Secretary's  decision  was 
primarily  based  on  the  results  of  the  latest  operational 
tests  which  showed  the  Sergeant  York  achieved  only  marginal 
improvement  over  other  available  systems  in  the  Army's 
inventory. 

The  Army  had  chosen  an  accelerated  acquisition  strategy 
for  the  procurement  of  the  Sergeant  York.  The  intention  of 
the  Army  was  to  field  the  weapon  system  as  quickly  as 
possible  with  substantial  cost  savings.  The  Army  recognized 
the  accelerated  strategy  carried  higher  than  usual  cost  and 
technical  risks.  First,  the  decision  to  schedule  less 
testing  prior  to  production  increased  the  possibility  of 
producing  a  weapon  system  with  unproven  technical 
capabilities.  Secondly,  the  accelerated  strategy  increased 
the  likelihood  of  cost  growth  due  to  lack  of  slack  time  in 
the  schedule.  The  Army  sought  to  minimize  the  cost  risks  by 
providing  safeguards,  such  as  a  requirement  for  the 
contractors  to  use  proven,  mature  subsystems  and  by 
including  more  extensive  than  usual  warranty  provisions 
(GAO,  1986:  3). 

Up  to  the  point  of  termination,  the  Army  had 
apparently  succeeded  in  controlling  costs  (GAO,  1986:  4) . 
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However,  the  desired  schedule  was  not  maintained  and  the 
technical  performance  was  deficient  in  several  key  areas. 

The  decision  to  use  mature  components  had  contributed  to  the 
lack  of  cost  growth.  Unfortunately,  this  decision  also 
contributed  to  the  significant  technical  difficulties  which 
arose  during  attempts  to  integrate  the  subsystems  into  a 
workable  unit.  The  end  results  of  such  difficulties  were 
delays  in  testing  and  the  production  of  prototypes  with 
less-than  mature  technological  capabilities. 

The  use  of  an  accelerated  strategy  was  not,  in  itself, 
the  cause  of  the  Sergeant  York's  demise.  The  program's 
cancellation  was  a  result  of  the  Army' s  decision  to  not 
allow  additional  time  to  resolve  technical  problems  in  the 
system's  targeting  subsystem.  When  technology  did  not 
mature  fast  enough  to  maintain  schedule,  the  Army  chose  to 
exercise  fixed-price  production  options  for  the  sake  of  cost 
control  and  disregarded  the  risks  the  weapon  system' s 
technical  difficulties  posed  (GAO,  1986:  6-7).  When  the 
technical  shortcomings  were  finally  revealed  during  live- 
fire  tests  in  mid-1985,  the  Secretary  of  Defense  was  left 
with  little  choice  but  to  cancel  the  program.  Sergeant  York 
had  demonstrated  less  than  adequate  performance  and  was  due 
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for  the  first  major  production  contract  in  January  1986, 
only  five  months  away  from  the  cancellation  decision. 

Although  the  Army  most  likely  understood  the 
"uncertainty"  surrounding  the  unproven  technology  in  the 
Sergeant  York,  I  don't  believe  they  understood  the  "damage" 
that  could,  and  did,  result  from  the  system's  inability  to 
meet  the  specified  level  of  performance.  Additionally,  the 
Army  did  not  develop  adequate  "safeguards",  such  as  slack 
time  in  the  schedule,  to  help  mitigate  the  consequences  of 
risk. 

Like  the  Army's  Sergeant  York,  the  Air  Force's  Advanced 
Medium-Range  Air-to-Air  Missile  (AMRAAM)  program  was  a 
technically  risky  venture  with  a  compressed  acquisition 
schedule.  The  AMRAAM  program  arose  from  an  Air  Force  and 
Navy  need  for  a  high-speed  missile  capable  of  beyond-visual- 
range  (BVR)  interception  of  enemy  aircraft  (Mayer,  1993: 

15) .  This  "launch  and  leave"  capability  would  involve  an 
active  guidance  and  control  system  that  would  allow  the 
missile  to  track  and  destroy  targets  on  its  own,  without 
assistance  from  the  pilot  in  the  launch  aircraft.  Through 
the  use  of  advancing  technologies,  the  AMRAAM  was  envisioned 
as  having  significant  performance  improvements  over  its 
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predecessor,  the  AIM-7,  while  maintaining  lower  life-cycle 


costs. 

From  very  early  in  the  program,  AMRAAM  was  exposed  to  a' 
high  level  of  external  demands.  In  fact,  congressional 
pressure  to  shorten  the  acquisition  cycle  and  maximize 
competition  was  a  key  factor  in  how  the  Air  Force  structured 
the  program  (Mayer,  1993:  53).  Additionally,  in  an  effort 
to  compete  with  a  proposed  decision  to  upgrade  the  Sparrow 
(AIM-7)  missile,  AMRAAM  program  officials  oversold  the 
program  in  terms  of  the  cost  and  schedule  requirements  for 
development  and  production  (Mayer,  1993:  52). 

To  a  great  degree,  the  technical  risks  associated  with 
the  development  of  the  AMRAAM  were  known  and  understood. 

The  technology  was  still  evolving,  but  was  predictable 
enough  to  develop  realistic  specifications  and  timetables. 
Problems  arose  when  forces  external  to  the  program  imposed 
"hazards"  for  which  the  Air  Force  had  not  prepared  adequate 
"safeguards".  Congressional  mandates  to  utilize  a 
compressed  schedule  and  to  second-source  production  options 
were  not  consistent  with  the  degree  of  technical  risk 
associated  with  the  program  (Mayer,  1993:  53).  To  the  Air 
Force's  credit,  the  end  result  was  a  fully  capable  weapon 
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system.  But  the  results  of  poor  risk  management  are 
demonstrated  by  an  initial  operating  capability  (IOC)  slip 
t  of  over  five  years  and  a  final  unit  cost  of  4  to  6  times 

greater  than  initial  estimates  (Mayer,  1993:  18,52). 


HOW  THE  FEDERAL  GOVERNMENT  MANAGES  RISK 

As  mentioned  earlier,  risk  is  primarily  a  function  of 
the  uncertainty  of  an  event  occurring  and  the  possible 
consequences  associated  with  that  uncertainty.  These  two 
elements,  uncertainty  and  consequence,  are  the  cornerstones 
of  risk  management. 

Within  the  DoD,  there  are  five  facets  of  risk 
associated  with  every  project — technical,  supportability, 
programmatic,  cost,  and  schedule  (DSMC,  1989:  3-3).  The 
first  three  facets — technical,  supportability,  and 
programmatic — are  related  to  the  uncertainty  of  the  project. 
The  final  two — cost  and  schedule — reflect  the  possible 
consequences  associated  with  the  failure  to  meet  established 
goals . 

Technical  risks  are  those  risks  associated  with 
evolving  a  new  design  to  achieve  a  higher  level  of 
performance,  or  to  maintain  a  constant  level  of  performance 
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under  new  constraints  (DSMC,  1989:  3-3) .  Many  of  the 
systems  procured  by  the  DoD  are  expected  to  achieve  a  high 
level  of  performance.  Often,  the  technical  requirements 
necessary  to  achieve  high  levels  of  performance  involve 
projections  about  the  future  state  of  technology.  The  level 
of  uncertainty  associated  with  the  future  projections  of 
technology  in  turn  determines  the  level  of  risk  associated 
with  the  program. 

Programmatic  risks  are  the  risks  associated  with 
acquisition  and  utilization  of  resources  which  may  be  beyond 
the  control  of  the  program  manager  (DSMC,  1989:  3-5) .  They 
are  often  a  function  of  the  business  environment  and 
generally  are  not  directly  related  to  improving  state-of- 
the-art.  The  most  common  form  of  programmatic  risk  in  DoD 
acquisition  is  the  uncertainty  of  political  decisions  made 
at  levels  of  authority  above  the  program  manager. 

Supportability  risks  are  associated  with  the  fielding 
and  maintenance  of  the  developed  systems  (DSMC,  1989:  3-6). 
The  technical  complexity  of  today's  systems  places  a  heavy 
burden  on  the  maintenance  and  logistics  requirements  to 
support  those  systems.  Additionally,  the  multitude  of 
physical  environments  in  which  the  systems  must  operate  pose 
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a  great  challenge  for  those  who  must  design  a  usable  and 


maintainable  product. 


Table  2-1:  Sources  of  Risk  (DSMC,  1989:  3-4) 


TYPICAL 

TYPICAL 

TYPICAL 

TYPICAL 

TYPICAL 

TECHNICAL  RISK 

PROGRAMMATIC  RISK 

SUPPORTABILITY  RISK 

COST  RISK 

SCHEDULE  RISK 

SOURCES 

SOURCES 

SOURCES 

SOURCES 

SOURCES 

Physical  Properties 

Material  Availability 

Reliability 

Sensitivity  to 

Sensitivity  to 

Technical  Risk 

Technical  Risk 

Material  Properties 

Personnel  Availability 

Maintahability 

Sensitivity  to 

Sensitivity  to 

Radiation  Properties 

Personnel  Skills 

Training 

Programmatic  Risk 

Programmatic  Risk 

Testing/Modeling 

Safety 

Operations  and  Support 

Sensitivity  to 

Sensitivity  to 

Equipment 

Supportability  Risk 

Supportability  Risk 

Integration/Interface 

Security 

Manpower  Considerations 

Sensitivity  to 

Sensitivity  to 

Software  Design 

Environmental  Impact 

Facility  Considerations 

Schedule  Risk 

Cost  Risk 

Safety 

Communication  Problems 

Interoperability 

Overhead/G&A  Rates 

Degree  of  Concumency 

Requirements  Change 

Labor  Strikes 

Considerations 

Estimating  Error 

Number  of  Critical 

Path  Items 

Fault  Detection 

Requirement  Changes 

Transportability 

Estimating  Error 

Operating  Environment 

Political  Advocacy 

System  Safety 

Slack  Time 

Proven/Unproven 

Contractor  Stability 

Technical  Data 

Technology 

Funding  Profile 

System  Complexity 

Regulatory  Changes 

Unique/Special  Resources 

The  final  two  facets  of  risk — cost  and  schedule — can  be 
seen  as  a  reflection  of  how  well  DoD  managers  have  dealt 
with  the  uncertainties  associated  with  technical, 
programmatic  and  supportability  risks.  Failure  to 
adequately  account  for  these  uncertainties  is  often 
reflected  as  cost  overruns  and  schedule  extensions.  There 
is  a  long  history  of  cost  and  schedule  growth  in  federal 
acquisition.  In  an  era  of  limited  DoD  budgets,  the  impacts 
associated  with  cost  overruns  and  schedule  extensions  are 
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particularly  disconcerting  (see  Sergeant  York  and  AMRAAM 
examples  above) . 

For  DoD  acquisition  managers,  risk  involves  three 
steps —  risk  assessment,  risk  analysis,  and  risk  handling 
(DSMC,  1989:  4-2).  Risk  assessment  is  the  process  of 
identifying  and  describing  the  sources  of  risk  within  the 
program.  Risk  analysis  is  the  process  of  developing  tools 
to  assist  in  determining  the  uncertainties  and  consequences 
associated  with  each  source  of  risk.  The  third  step,  risk 
handling,  is  the  process  of  developing  "safeguards"  to 
account  for  the  sources  of  risk. 

As  mentioned  at  the  beginning  of  this  section,  the  risk 
elements  of  uncertainty  and  damage  are  the  foundation  of 
risk  management  within  the  federal  government.  Through  the 
use  of  risk  assessment  and  risk  analysis  techniques,  the 
government  identifies  sources  of  risk  within  a  program  and 
then  determines  the  uncertainty  and  damage  associated  with 
the  occurrence  of  "risky"  events.  Then,  safeguards  are 
developed  to  minimize  the  risk  of  damaging  consequences  like 


cost  growth. 


HOW  THE  GOVERNMENT  CONTRACTOR  MANAGES  RISK 


Government  contractors  often  face  difficult  choices 
when  deciding  how  to  position  themselves  with  regard  to  risk 
in  a  program.  In  a  competitive  situation,  the  contractor 
must  often  bid  aggressively  in  hopes  of  winning  a  contract 
(Mayer,  1993:  8).  If  a  company  is  too  optimistic  about  its 
chances  of  meeting  a  cost  target,  however,  it  may  expose 
itself  to  significant  financial  risk. 

Like  the  government,  effective  risk  management  for  the 
contractor  begins  with  an  independent  assessment  of  the 
technical  and  financial  risks  associated  with  a  program. 
Based  on  the  Government's  proposed  requirements,  the 
contractor  will  assess  the  degree  of  technical  risk 
associated  with  performance  or  design  criteria,  as  well  as 
the  schedule  risk  associated  with  maintaining  the  desired 
timeline  of  events.  In  the  risk  assessment  process,  the 
contractor  is  making  determinations  about  the  "uncertainty" 
involved  in  their  ability  to  achieve  the  requirements  of  the 
program. 

In  addition  to  assessing  the  technical  risks  associated 
with  each  program,  the  contractor  must  also  assess  the  level 
of  "programmatic"  risk.  Not  unlike  the  government,  the 
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primary  influence  to  programmatic  risk  for  the  contractor  is 
political  in  nature.  Instead  of  responding  to  congressional 
inquiries  and  inputs,  however,  the  contractor  must  answer  to- 
stockholders  and  lien-holders.  As  a  business  institution, 
the  contractor  must  focus  on  achieving  an  acceptable  rate  of 
return  on  their  investment  in  order  to  build  capital  and 
remain  solvent  (Mayer,  1993:  9) .  In  doing  so,  the 
contractor  must  influence  the  stakeholders  of  the 
corporation  that  the  level  of  risk  taken  in  bidding  on  a 
program  is  worthwhile  in  comparison  to  the  possible  rewards 
associated  with  success.  This  form  of  risk  assessment  is 
similar  to  conducting  a  cost-benefit  analysis  of 
alternatives . 

Even  though  the  contractor  is  concerned  with  the 
"uncertainty"  involved  in  risky  behavior,  there  is  evidence 
to  support  their  primary  concern  being  the  impact  associated 
with  the  failure  or  success  emanating  from  risky  decisions. 

A  recent  study  on  the  managerial  perspective  of  taking  risks 
found  that  over  95%  of  the  executives  surveyed  primarily 
associated  the  concept  of  risk  with  detrimental  outcomes 
(Shapira,  1995:  45).  The  executives  tended  to  downplay  or 
ignore  the  probability  of  risk,  instead  focusing  on  the 
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impact  of  their  decisions.  One  executive  pointed  out  that 
the  uncertainty  associated  with  two  good  outcomes  is  not 
risky  at  all  (Shapira,  1995:  44) .  Others  indicated  their 
decisions  involving  risk  depended  on  the  estimates  of  the 
worst  possible  outcome.  "Only  if  the  negative  outcome  was 
tolerable  did  they  consider  the  alternative  and  eventually 
looked  at  the  possible  opportunities  involved  in  making  the 
decision"  (Shapira,  1995:  46).  While  these  managers  were 
totally  aware  of  the  positive  sides  to  taking  risks  and  the 
expected  value  approach  of  classical  decision  theory,  they 
were  clearly  more  influenced  by  the  impact  associated  with 
their  decisions. 

At  some  point,  the  prospect  of  encountering  risky 
decisions  is  inevitable.  In  order  to  alleviate  the  possible 
impacts,  the  contractors,  like  the  government,  develop 
"safeguards"  to  reduce  risk.  One  "safeguard"  the  contractor 
uses  to  avoid  financial  risk  is  through  the  creation  of  a 
reserve  budget.  This  budget  is  an  amount  of  money  set  aside 
by  management  to  help  provide  additional  funding  in  the 
result  of  poor  initial  estimates.  The  reserve  budget  is  not 
unlike  a  personal  savings  one  would  have  available  for  use 
in  the  event  of  an  unexpected  repair  bill  to  their  car  or 
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home.  The  amount  of  personal  savings  held,  in  this  case, 
would  be  subject  to  an  individual  assessment  of  the 
likelihood  of  needing  to  make  repairs  (uncertainty)  and  the 
possible  costs  associated  with  the  repairs  (impact) . 
Similarly,  the  amount  of  budget  created  by  the  defense 
contractor  is  a  reflection  of  the  uncertainty  and  impact 
contained  in  their  risk  assessment  (Gould,  1995) . 

In  summary,  the  contractor  makes  an  independent 
judgment  of  the  risks  involved  in  defense  programs.  To 
assist  in  decision  making,  the  contractor  assesses  the 
uncertainty  of  meeting  technical,  schedule,  and  cost 
parameters  and  the  impact  which  may  result  from  a  failure  to 
meet  those  parameters.  Then,  with  an  understanding  of  the 
uncertainties  and  their  impacts,  the  contractor  develops 
safeguards  in  order  to  minimize  the  overall  risk  of  the 
decision. 

SETTING  UP  TESTABLE  RELATIONSHIPS 

The  elements  of  risk — uncertainty,  consequence, 
safeguard — are  constants  in  DoD  acquisition.  We  have  seen 
how  a  proactive  stance  towards  risk  management  helps  to 
minimize  the  consequences  inherent  in  risky  decisions.  One 
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of  the  keys  to  effective  risk  management  has  been  shown  to 
involve  an  accurate  assessment  of  the  actual  risks  involved. 
*  ,  Failure  to  accurately  assess  the  risk  leads  to  consequences 

i*  which,  most  often,  are  damaging  in  nature. 

One  of  the  most  damaging  results  of  poor  risk 
management  within  the  DoD  is  the  issue  of  cost  growth.  The 
average  cost  overrun  on  defense  contracts  has  been  estimated 
at  40  percent  (Gansler,  1989:  4).  Considering  the  enormity 
of  acquisition  expenditures  within  the  DoD,  this  estimate 
could  result  in  dollar  overruns  of  $37B  per  year. 

Extensive  research  has  been  conducted  to  determine  the 
cause  of  cost  growth  in  Federal  acquisition  programs 
(Drezner,  1995; Fletcher  &  Young,  1994 ; Singleton,  1991).  No 
one,  however,  has  developed  a  single  definitive  explanation 
for  cost  overruns.  A  1993  Rand  study  set  out  to  identify 
factors  affecting  cost  growth  (Drezner,  et.al,  1995:  xi) . 
They  examined  aspects  ranging  from  program  length,  size,  and 
maturity  to  management,  budget,  and  political  decisions. 

^  Their  overall  observation:  "the  problem  of  cost  growth  does 

not  have  a  'silver  bullet'  policy  response"  (Drezner,  et.al, 

m 

1995:  xiii) .  In  other  words,  many  factors  affect  the 
problem  of  cost  overruns,  yet  no  one  factor  can  individually 
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be  labeled  as  the  defining,  overarching  precedent  to  cost 
growth. 

In  an  effort  to  find  better  methods  of  controlling  and 
predicting  acquisition  costs,  the  DoD  implemented  a  set  of 
criteria  known  as  the  Cost/Schedule  Control  Systems  Criteria 
(C/SCSC) .  DoD  instruction  5000.2  requires  the  use  of  C/SCSC 
for  research,  development,  test  and  evaluation  contracts  of 
$60  million  or  more  or  procurement  contracts  valued  at  $250 
million  or  more  (DoD,  1991:  11B2) .  As  outlined  in  DoDI 
5000.2,  "the  purpose  of  cost/schedule  control  systems 
criteria  is  to  provide  contractor  and  the  Government  program 
managers  with  accurate  data  to  monitor  execution  of  their 
program"  (DoD,  1991:  llBl) .  The  intention  is  to  standardize 
evaluation  and  reporting  procedures  so  that  accurate  and 
prompt  management  decisions  can  be  made  regarding  the 
achievement  of  cost,  schedule  and  technical  elements  of  the 
program. 

Despite  the  detailed  efforts  to  monitor  and  control 
costs,  Drezner  indicates  a  lack  of  substantial  improvement 
in  average  cost  growth  over  the  past  30  years  (Drezner, 
et.al,  1995:  xiii) .  In  a  report  on  weapon  systems  cost 
trends,  James  Wiggins  of  the  General  Accounting  Office 


36 


stated  "the  practice  of  making  unrealistically  low  initial 
cost  estimates  has  been  the  major  contributor  to  weapon 
systems  cost  growth"  (Wiggins,  1988:  16).  Similarly,  a  1993 
study  by  Dr.  David  Christensen  confirms  that  poor  initial 
estimates  are  a  major  cause  of  cost  overruns  and  that 
"recoveries  from  cost  overruns  on  defense  contracts  are 
highly  improbable"  (Christensen,  1993:  47).  Poor  estimates 
alone,  however,  do  not  account  for  the  enormity  of  cost 
overruns  experienced  in  today's  acquisition  community. 

Cost  estimating  is,  however,  one  of  the  steps  used  by 
the  DoD  when  conducting  risk  assessment.  Although  much 
research  has  been  accomplished  to  investigate  the  causes  of 
cost  growth,  little  has  been  conducted  to  address  how  poor 
risk  assessment  affects  cost  growth.  The  intent  of  this 
study  is  to  investigate  the  effect  of  inaccurate  risk 
assessment  on  the  government's  ability  to  achieve  cost 
targets . 

Each  party  to  the  contract,  the  government  and  the 
contractor,  conducts  an  independent  assessment  of  the  risks 
associated  with  a  contract.  This  risk  assessment  is  a 
reflection  of  their  perception  of  the  level  of  risk  faced. 
Poor  risk  assessments  can  lead  to  consequences,  most  of 
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which  are  damaging  in  nature.  Poor  risk  assessments  can 
also  lead  to  consequences  which,  on  the  surface,  appear  to 
be  beneficial.  For  instance,  an  assessment  of  risk  which  is' 
higher  than  the  actual  risk  could  result  in  an  overall  cost 
underrun.  While  this  initially  appears  to  be  a  desirable 
outcome,  it  in  fact  means  funds  were  under-utilized, 
possibly  to  the  detriment  of  other  programs. 

Hypotheses 

Inaccurate  risk  assessments  can  lead  to  undesirable 
consequences.  One  of  these  consequences  is  the 
aforementioned  inability  to  achieve  cost  targets.  If  both 
the  contractor's  and  the  government's  perceptions  of  risk 
are  in  alignment  with  the  actual  risk  involved  in  a 
contract,  then  we  would  expect  minimal  impact.  If,  however, 
either  or  both  parties  fail  to  accurately  assess  the  actual 
risk,  then  we  would  expect  consequences,  such  as  cost 
overrun/underrun.  Consequently,  it  may  be  argued  that  cost 
overrun/underrun  is,  in  part,  a  function  of  the  alignment 
between  the  actual  risk  and  each  parties  perceptions  of  the 
risk.  In  instances  were  risk  is  underestimated,  the 
expectation  is  that  the  contract  will  experience  a  cost 
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overrun,  because  the  parties  have  failed  to  adequately 
provide  "safeguards"  to  help  minimize  the  consequences  of 
the  actual  risk  level.  Conversely,  if  risk  is 
overestimated,  the  expectation  is  that  managers  have  under¬ 
utilized  their  funds  and  the  contract  will  experience  a  cost 
underrun. 

Hypothesis  (1)  -  Cost  underruns  will  be  highest,  if  both  the 
government  and  the  contractor  perceptions  of  risk  are  higher 
than  the  actual  risk. 

Hypothesis  (2)  -  Cost  overruns  will  be  highest,  if  both  the 
government  and  the  contractor  perceptions  of  risk  are  lower 
than  the  actual  risk. 
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III.  Methodology 


OVERVIEW 

The  purpose  of  this  chapter  is  to  describe  the 
methodology  used  to  examine  the  relationship  between  risk 
alignment  and  cost  growth  in  DoD  acquisition.  The  chapter 
begins  with  a  description  of  the  population  of  interest  and 
the  subset  of  data  taken  from  that  population  to  be  used  as 
the  sample  for  this  study.  The  validity  and  measurement 
techniques  for  the  dependent  variable  (cost 
overrun/underrun)  and  independent  variables  (contract  type, 
contractor  management  reserve,  and  acquisition  phase)  are 
then  established.  The  chapter  concludes  with  the 
development  of  the  analytical  procedure  (ANOVA)  and  model 
used  to  investigate  the  hypotheses  presented  in  Chapter  II. 

POPULATION  OF  INTEREST  AND  THE  SAMPLE 

The  population  of  interest  was  all  acquisition 
contracts  within  the  Federal  Government.  Cost  performance 
data  of  a  subset  of  that  population  is  reported  in  the 
Defense  Acquisition  Executive  Summary  (DAES)  database.  The 
DAES  database  contains  data  on  completed  and  on-going  DoD 
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contracts  since  1977  (Christensen,  1993:  45) .  The  data  is 


summarized  in  the  form  of  Cost  Performance  Reports  (CPRs) 
and  submitted  to  the  Office  of  the  Under  Secretary  of 
Defense  for  Acquisition  &  Technology  (OUSD(A&T))  on  a 
quarterly  basis  (DoD,  1991) .  The  database  contains 
contracts  from  all  three  military  services  and  covers  a  wide 
variety  of  contract  types  and  end-items. 

The  sample  consisted  of  all  contracts  in  the  database 
that  are  complete  or  nearly  completed  and  that  contain  the 
information  necessary  to  define  the  variables  of  interest. 
For  the  purposes  of  this  study,  nearly  complete  was  defined 
as  contracts  that  were  past  the  85%  completion  point.  It 
has  been  shown  that  once  a  contract  reaches  the  85% 
completion  point,  over  95%  of  the  total  costs  will  have  been 
incurred  (Wilson,  1992:  38).  Percent  complete  (PC)  is  a 
function  of  the  Budgeted  Cost  of  Work  Performed  (BCWP)  and 
the  Budget  at  Completion  (BAC)  and  was  defined  as  follows: 

PC  =  BCWP /BAC. 

MEASUREMENT  OF  THE  DEPENDENT  VARIABLE 

The  dependent  variable  of  interest  for  this  study  was 
the  final  cost  variance  of  contracts  in  the  DAES  database. 
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At  contract  completion,  the  overrun  or  underrun  was  the 
difference  between  the  total  budget  for  all  the  work  on 
contract,  termed  the  "Budget  at  Completion"  (BAG) ,  and  final  ' 

actual  cost  of  the  work  performed  (ACWP)  (Christensen,  1993:  ^ 

44) .  In  the  case  of  an  overrun,  the  money  actually  paid  to 
contractor  for  completion  of  the  work  (ACWP)  was  greater 
than  the  monies  budgeted  for  that  work  (BAC) .  Conversely, 
an  underrun  exists  if  the  ACWP  was  lower  than  the  final  BAC. 

For  the  purposes  of  this  study,  a  numerical  rating  of  cost 
performance  was  assigned  through  the  following  formula: 

Final  Cost  Overrun/Underrun  (FCO/U)  =  ACWP/BAC. 

By  this  definition,  contracts  exhibiting  an  FCO/U  greater 
than  one  experienced  a  cost  overrun,  while  those  contracts 
with  an  FCO/U  less  than  one  resulted  in  a  cost  underrun.  A 
relative  indication  of  cost  overrun/underrun  was  thus 
established  by  the  degree  to  which  the  FCO/U  for  each 
contract  was  above  or  below  unity. 
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MEASUREMENT  OF  THE  INDEPENDENT  VARIABLES 


Three  independent  variables  were  necessary  to 
investigate  the  relationship  between  risk  alignment  and’  the 
dependent  variable — cost  performance.  The  variables  needed 
were:  a  measure  of  the  actual  risk,  a  measure  of  the 
government's  perception  of  risk,  and  a  measure  of  the 
contractor's  perception  of  risk. 

Actual  Risk 

A  measure  of  the  actual  risk  encountered  in  a  contract 
was  exhibited  by  the  phase  of  development  of  the  system. 
During  the  research  and  development  (R&D)  phase  of  a  system, 
the  requirements  are  not  fully  refined  and  the  desired  level 
of  technology  has  not  fully  matured.  For  these  and  other 
reasons,  the  uncertainty  associated  with  the  state  of  the 
final  design  tends  to  create  a  higher  level  of  risk  in 
meeting  objectives.  Ideally,  once  a  program  transitions 
into  the  production  phase,  the  system  has  a  stable  design. 

At  this  point,  the  manufacturing  and  production  processes 
have  been  validated  and  the  system  produced  has  demonstrated 
the  ability  to  meet  contractual  specifications  and  satisfy 
minimum  operational  performance  requirements  (Przemieniecki, 
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1993:  27).  The  lower  level  of  uncertainty  during  production 
tends  to  indicate  a  shift  in  the  actual  risk  of  meeting 
objectives . 

The  contracts  in  the  DAES  database  were,  therefore, 
separated  into  low  and  high  risk  categories  based  on  the 
phase  of  development.  Contracts  pertaining  to  the  R&D  phase 
of  development  were  coded  as  high  risk,  while  contracts  used 
during  production  were  coded  as  low  risk. 

The  Government' s  Perception  of  Risk 

The  government's  perception  of  cost  risk  was  reflected 
in  the  selection  of  contract  type.  The  government  chooses  a 
contract  type  it  believes  accurately  reflects  the  certainty 
or  uncertainty  of  achieving  cost  targets.  If  the  government 
believes  there  is  a  high  certainty  that  the  contractor  can 
achieve  program  goals  and  still  maintain  adequate  cost 
control,  then  they  tend  to  select  a  fixed-price  type 
contract.  By  doing  so,  the  government  is  relieving  itself 
of  most  of  the  burden  of  cost  overruns  by  requiring  the 
contractor  to  complete  the  contract  for  a  fixed  price. 
Conversely,  by  selecting  a  cost  reimbursable  type  of 
contract,  the  government  is  suggesting  that  there  is  high 
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uncertainty  in  the  contract,  and,  therefore  the  risk  of  cost 


escalation  is  also  high.  In  this  instance,  the  government 
assumes  the  risk  of  cost  overruns  by  agreeing  to  reimburse 
the  contractor,  within  set  limits,  for  the  work  associated 
with  completing  the  contract. 

Similarly,  the  type  of  contract  selected  will  coincide 
with  the  certainty  or  uncertainty  associated  with  the 
technical  risks  in  the  contract.  A  well  defined  system  with 
complete  specifications  will  usually  lend  itself  to  be 
procured  most  effectively  with  a  fixed-price  arrangement  due 
to  the  increased  certainty  of  the  contractor's  ability  to 
meet  the  requirements.  Conversely,  if  the  technical 
requirements  are  not  so  well  defined,  there  is  a  high  degree 
of  uncertainty  in  the  contractor's  ability  to  achieve  the 
specified  level  of  performance  and  still  meet  program 
objectives.  In  this  instance,  the  government  assumes  the 
burden  of  overrun  risk  through  the  selection  of  a  cost- 
reimbursable-type  contract. 

Summarily,  we  can  see  how  the  government's  perception 
of  the  level  of  risk  is  captured  through  the  selection  of  a 
particular  contract  type.  The  variable  pertaining  to  the 
government's  perception  of  risk  was,  therefore,  the  contract 
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type  shown  in  the  DAES  database.  Fixed-price  type  contracts 
indicated  a  low  level  of  perceived  risk,  while  cost- 
reimbursable  type  contracts  reflected  a  high  level  of  risk. 


The  Contractor's  Perception  of  Risk 

An  indication  of  the  contractor's  perception  of  risk 

was  the  level  of  reserve  budget,  or  management  reserve  (MR) , 

allocated  to  the  contract.  Management  reserve  (MR)  is  part 

of  the  10th  criterion  of  the  Cost  Schedule  Control  Systems 

Criteria  (C/SCSC)  (DoD,  1987)  The  C/SCSC  allow  the 

contractor,  after  contract  award,  to  budget  a  management 

reserve  into  the  program.  This  MR  budget  is  held  in  reserve 

by  the  contractor  program  manager  and  is  intended  to  be  used 

for  management  control  purposes. 

In  most  major  acquisition  contracts, 
particularly  in  the  development  phase, 
there  is  considerable  uncertainty 
regarding  the  timing,  contractor  work 
breakdown  structure  (CWBS)  elements 
involved,  or  magnitude  of  difficulties. 

The  C/SCSC  permit  the  use  of  a 
management  reserve  provided  that 
adequate  identification  and  controls  are 
maintained.  (DoD,  1987:  3-10,  emphasis 
added) 


A  1995  thesis  by  Kevin  Gould  set  out  to  provide  a  field 
expert  evaluation  of  the  purpose  and  development  of 
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management  reserve  (Gould,  1995).  According  to  Gould's 


investigation  of  former  studies,  a  primary  use  of  the 
<  contractor's  management  reserve  is  "to  provide  an  adequate 

,  budget  for  in-scope,  unanticipated  performance  requirements 

that  will  impact  the  future  effort"  (Gould,  1995:  3).  As 
part  of  the  research  effort,  Gould  developed  a  series  of 
questions  relative  to  management  reserve  and  then 
interviewed  ten  different  government  and  contractor  managers 
with  extensive  experience  in  the  field. 

Overwhelmingly,  respondents  indicated  MR  is  a 
reflection  of  the  contractor's  assessment  of  the  level  of 
risk  associated  with  the  program,  in  general,  and  the 
contract  specifically  (Gould,  1995:  34-35).  Gould's 
analysis  led  him  to  the  conclusion  that  "risks  are  used  to 
assist  in  the  computation  of  an  appropriate  level  of 
management  reserve  budget"  (Gould,  1995:  36) . 

Another  part  of  Gould' s  research  indicated  that  the 
average  level  of  MR  was  7.5%  of  the  total  budget  (Gould, 
1995:  9).  Using  this  figure  as  a  crude  breaking  point, 
contracts  in  the  database  with  an  initial  MR  greater  than 
7.5%  were  categorized  as  being  high  risk,  as  perceived  by 
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the  contractor.  Conversely,  contracts  with  a  reported 
initial  MR  of  less  than  7.5%  were  categorized  as  low  risk. . 

DESIGN  OF  THE  MODEL 

The  hypotheses  presented  in  Chapter  II  were  tested  by 
examining  the  final  cost  overrun/underrun  (FCO/U)  for  each 
of  the  possible  alignments  of  the  variables  described  above. 
The  possible  combinations  are  presented  in 
Table  3-1. 


Table  3-1:  Risk  Alignment  Combinations 


Combination  Number 

■ 

2 

3 

4 

5 

6 

7 

8 

Actual  Risk 

H 

H 

H 

H 

L 

H 

■ 

L 

Government's  Risk 
Perception 

■ 

L 

H 

H 

■ 

H 

H 

H 

Contractor's  Risk 
Perception 

■ 

H 

■ 

H 

L 

H 

L 

H 

Combination  #1  above  represents  the  alignment  described 
in  Hypothesis  (2) ,  whereby  an  underestimation  of  risk  by 
both  parties  should  result  in  the  highest  level  of  cost 
overrun.  Likewise,  combination  #8  above  represents  the 
alignment  of  Hypothesis  (1) .  The  overestimation  of  risk  by 


both  parties  should  result  in  the  highest  levels  of  cost 
underrun.  Combinations  #4  and  #5  above  reflect  an  accurate 
risk  assessment  by  both  parties.  These  two  combinations 
represent  an  ideal  scenario,  whereby  the  level  of  cost 
overrun/underrun  should  be  minimal. 

The  Analytical  Model 

An  Analysis  of  Variance  (ANOVA)  technique  was  utilized 
to  determine  if  the  variance  in  FCO/U  was  significant 
between  the  alignment  combinations  shown  in  Table  3-1. 

Using  the  Statistix  software  package,  a  one-way  ANOVA 
provided  results  on  the  between-groups  effect  and  the 
equality  of  within-group  variances  (Analytical  Software, 
1994:  130).  Additionally,  the  ANOVA  allowed  for  a 
comparison  of  means  option,  which  was  used  to  compare  the 
mean  values  of  FCO/U  for  each  of  the  eight  risk  alignment 
combinations.  All  tests  were  conducted  at  the  p-value=.05 
significance  level.  The  ANOVA  technique  was  selected  for 
use  in  this  study  because  of  its  ability  to  quickly  analyze 
multiple  levels  of  combinations.  Also,  the  ANOVA  process 
did  not  pose  the  restriction  of  equal  sample  sizes  within 
the  combination  levels. 
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IV.  Results 


OVERVIEW 

The  purpose  of  this  chapter  is  to  present  the  results 
of  the  statistical  analysis  described  in  Chapter  III. 
Descriptive  statistics  and  variable  correlations  of  the 
sample  are  followed  by  the  ANOVA  results  and  comparison  of 
mean  values  of  the  risk  alignment  combinations.  The  risk 
alignment  combinations  and  their  associated  combination 
references  are  presented  again  in  Table  4-1. 


Table  4-1:  Risk  Alignment  Combinations 


Combination  Number 

1 

2 

3 

4 

5 

6 

7 

8 

Actual  Risk 

H 

H 

H 

H 

L 

L 

L 

L 

Government's  Risk 
Perception 

L 

L 

H 

H 

L 

L 

H 

H 

Contractor's  Risk 
Perception 

L 

H 

L 

H 

L 

H 

L 

H 

DESCRIPTIVE  STATISTICS  OF  THE  SAMPLE 

Contracts  in  the  DAES  database  were  divided  into  eight 
combinations  in  accordance  with  the  risk  alignment  described 
in  Chapter  III.  The  frequency  of  each  risk  alignment 
combination  is  shown  in  Table  4-2  below. 
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Table  4-2:  Frequency  Distribution  of  Combinations 


COMBINATION 

NUMBER 

FREQUENCY 

FREQUENCY 

PERCENTAGE 

1 

17 

5.4 

2 

20 

6.4 

3 

46 

14.7 

4 

19 

6.1 

5 

137 

43.9 

6 

51 

16.3 

7 

19 

6.1 

8 

3 

1.0 

TOTAL 

312 

100.0 

The  combination  frequencies  exhibit  a  normal 
distribution  with  the  bulk  of  occurrences  (50%)  at 
combinations  4  &  5  and  a  diminishing  tendency  toward  the 
tails  (5.4%  for  combination  1  and  1%  for  combination  8). 


A  Pearson  correlation  table  of  the  dependent  and 
independent  variables  is  shown  in  Table  4-3. 


Table  4-3:  Variable  Correlations 


CTYPE 

FCOU 

MR 

FCOU 

-0.0151 

MR 

-0.0615 

0.0428 

PHASE 

0.5570 

-0.1543 

0.1284 
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The  correlations  appear  to  be  low  with  the  exception  of 
contract  type  (CTYPE)  to  procurement  phase  (PHASE) .  The 
value  of  .557  is  a  strong  correlation,  indicating  that 
changes  in  the  phase  of  procurement  tend  to  be  positively 
accompanied  by  changes  in  contract  type. 


ANOVA  RESULTS 

A  standard  analysis  of  variance  table  is  displayed  in 
Table  4-4. 


Table  4-4:  ANOVA  Statistics 


SOURCE 

DF 

SS 

MS 

F 

P 

BETWEEN 

7 

0.83867 

0.11981 

3.52 

0.0013 

WITHIN 

304 

10.3403 

0.03401 

TOTAL 

311 

11.1790 

The  reported  p-value  of  0.0013  suggests  a  substantial 
between-groups  effect.  This  indicates  there  is  reason  to 
believe,  at  the  .05  significance  level,  that  at  least  two  of 
combination  mean  values  differ  from  one  another. 
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Comparison  of  Mean  FCO/U  Values 


A  Bonferroni  pairwise  comparison  of  means  was  computed 
as  part  of  the  ANOVA  procedure.  The  results  are  shown  in 
Table  4-5. 


Table  4-5:  ANOVA  Pairwise  Comparison  of  Means 


COMBINATION 

MEAN 

FCO/U 

1 

4 

3 

5 

6 

2 

8 

1 

1.2264 

4 

3 

0.55 

5 

1.0268 

4.21* 

1.32 

1.01 

6 

1.0158 

4.08* 

1.42 

1.14 

0.36 

2 

1.0040 

3.66* 

1.39 

1.11 

0.52 

0.24 

8 

0.9866 

2.08 

0.87 

0.66 

0.37 

0.27 

0.15 

7 

0.9742 

4.10* 

1.87 

1.68 

1.17 

0.84 

0.50 

0.11 

Critical  T  Value  =  3.152  *p  value  <  .05 

The  combinations  in  Table  4-5  are  ordered  according  to 
the  mean  value  of  final  cost  overrun/underrun  (FCO/U) ,  with 
the  highest  value  appearing  first.  The  numbers  in  the  body 
of  the  table  are  the  t-statistics  for  each  combination  pair. 
A  t-statistic  higher  than  the  critical  value  of  3.152 
indicates  that  the  mean  values  of  FCO/U  for  that  pair  are 
statistically  different  at  the  .05  significance  level. 
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Combination  pairs  exhibiting  t-statistics  higher  than  the 
critical  value  are  indicated  by  the  bold  text. 

The  results  indicate  that  the  highest  FCO/U  value 
occurred  at  combination  number  one,  while  the  lowest  FCO/U 
value  occurred  at  combination  number  seven.  Note,  however, 
the  reported  mean  value  of  combination  number  eight  was  less 
than  1.0  and  that  the  mean  values  for  combinations  seven  and 
eight  were  not  statistically  different.  Note,  also,  that 
combination  number  one  exhibited  a  statistically  different 
mean  value  with  five  other  combinations — the  only  exceptions 
being  combinations  four  and  eight. 

One  of  the  assumptions  for  the  comparison  of  means 
ANOVA  test  is  that  the  within-group  variances  are  equal  for 
all  combinations.  Bartlett's  test  of  equal  variances  tests 
this  assumption.  The  results  are  shown  in  Table  4-6. 

Table  4-6:  Bartlett's  Test  of  Equal  Variances 


CHI-SQ 

DF 

P 

77.39 

7 

0.0000 

The  reported  p-value  of  0.0000  indicates  that  all  of 
the  combinations  do  not  exhibit  equal  variances.  The  equal 
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variances  assumption  necessary  for  this  ANOVA  is  therefore 


not  supported.  The  violation  of  equal  variances  between 
combinations  does  not  conclusively  invalidate  the  results  of 
the  ANOVA  comparison  of  means.  It  does,  however,  suggest 
that  the  resulting  differences  in  means  may  be  attributable 
to  variance  behavior  instead  of  the  actual  mean  FCO/U 
tendencies . 

Because  of  the  ANOVA  limitation  concerning  equal 
variances  among  combinations,  a  two-sample  t-test  was 
conducted  between  combinations  one  and  eight  and  the  other 
six  risk  alignment  combinations.  The  results  are  shown  in 
Table  4-7. 


Table  4-7:  T-Test  Comparison  of  Means 


COMBINATION 

MEAN 

p-  value 

Comb  1 

Comb  8 

1 

1.2264 

.0326* 

2 

1.0863 

.0258* 

.7237 

3 

1.0587 

.0212* 

.1629 

4 

1.0268 

.2505 

.5830 

5 

1.0158 

.0001* 

.3750 

6 

1.0040 

.0023* 

.5196 

7 

0.9866 

. 0112* 

.7735 

8 

0.9866 

.0326* 

- 
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The  reported  p-values  assume  either  unequal  or  equal 
combination  variances,  as  is  appropriate  for  each  individual 
pair.  The  p-values  shown  with  an  asterisk  indicate  the  mean 
values  of  FCO/U  for  the  two  combinations  are  statistically 
different  at  the  .05  significance  level. 

The  results  of  the  t-test  are  similar  to  those  reported 
in  the  ANOVA  comparison  of  means.  The  only  difference  being 
that  the  t-test  indicates  a  significant  difference  between 
mean  FCO/U  values  for  combinations  1  and  8. 

CHAPTER  SUMMARY 

Results  of  the  statistical  analysis  indicate 
significant  differences  exist  between  several  of  the  mean 
final  cost  underrun/overrun  (FCO/U)  values.  The  only 
significant  differences,  however,  were  between  combination  1 
and  five  of  the  other  risk  alignment  combinations  (see  Table 
4-5) .  Because  of  failed  equal  variances  assumption  of  the 
ANOVA  procedure,  a  t-test  had  to  be  conducted  to  confirm  the 
ANOVA  results.  The  results  of  the  ANOVA  were  supported  by 
the  t-test.  Additionally,  the  t-test  indicated  a 
statistically  significant  difference  existed  between  risk 
alignment  combinations  one  and  eight,  which  represented  the 
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difference  between  an  underestimation  of  risk  by  both  the 
government  and  the  contractor  and  an  overestimation  of  risk 
by  both  parties. 
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V.  Conclusions 


OVERVIEW 

Proper  risk  management  is  an  organized  means  of 
identifying  and  measuring  risk  and  developing,  selecting, 
and  managing  options  for  handling  these  risks.  It  involves 
understanding  the  danger  signals  that  may  indicate  a  project 
is  off  track  and  prioritizing  corrective  actions  as 
necessary.  Risk  management  is  not  a  separate  office 
activity,  but  rather  is  one  aspect  of  sound  technical 
management.  Proper  risk  management  implies  control  of 
possible  future  events,  and  is  proactive  rather  than 
reactive.  Proper  risk  management  will  reduce  not  only  the 
likelihood  of  an  unfavorable  event  occurring,  but  also  the 
magnitude  of  its  impact  (Kerzner,  1995:  880). 

The  recurring  premise  is  that  the  best  way  to  handle 
risky  situations  is  to  manage  them.  By  this,  I  mean  taking 
an  active  stance  in  identifying  potential  problem  areas  and 
then  preparing  a  necessary  course  of  action  to  alleviate  the 
damage  associated  with  risky  decisions. 

Cost  and  schedule  performance  on  a  contract  can  be  seen 
as  a  reflection  of  how  well  DoD  managers  have  dealt  with  the 
uncertainties  associated  with  risk.  Failure  to  adequately 
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account  for  these  uncertainties  is  often  reflected  as  cost 
overruns  and  schedule  extensions.  There  is  a  long  history 
of  cost  and  schedule  growth  in  federal  acquisition.  In  an 
era  of  limited  DoD  budgets,  the  impacts  associated  with  cost 
overruns  and  schedule  extensions  are  particularly 
disconcerting . 

The  Sergeant  York  and  AMRAAM  examples  discussed  in 
Chapter  II  reflect  the  possible  consequences  of  poor  risk 
management.  In  both  examples,  DoD  acquisition  managers 
either  failed  to  adequately  understand  the  "uncertainties" 
associated  with  the  risk  in  their  programs,  or  failed  to 
develop  the  necessary  "safeguards"  to  mitigate  the  damaging 
effects  of  a  risky  environment.  The  end  results  were  the 
cancellation  of  the  Sergeant  York  program  and  cost  overruns 

of  the  AMRAAM  by  an  estimated  4  to  6  times  the  initial 
estimates . 

As  part  of  the  risk  management  process,  each  party  to 
the  contract,  the  government  and  the  contractor,  conducts  an 
independent  assessment  of  the  risks  associated  with  a 
contract.  This  risk  assessment  is  a  reflection  of  their 
perception  of  the  level  of  risk  faced.  Poor  risk 
assessments  can  lead  to  consequences,  most  of  which  are 
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damaging  in  nature.  Poor  risk  assessments  can  also  lead  to 
consequences  which,  on  the  surface,  appear  to  be  beneficial. 
For  instance,  an  assessment  of  risk  which  is  higher  than  the 
actual  risk  could  result  in  an  overall  cost  underrun.  While 
this  initially  appears  to  be  a  desirable  outcome,  it  in  fact 
means  funds  were  under-utilized,  possibly  to  the  detriment 
of  other  programs. 

This  study  attempted  to  investigate  the  effect  of 
inaccurate  risk  assessment  on  the  government's  ability  to 
achieve  cost  targets.  The  organization  of  the  study  focused 
on  the  alignment,  or  lack  of  alignment,  between  each  party's 
perception  of  risk  and  the  actual  risk  encountered.  The 
actual  risk  was  believed  to  be  a  function  of  the  phase  of 
procurement  (high  risk  for  R&D  and  low  risk  for  production) . 
The  government  and  contractor  perceptions  of  risk  were  based 
on  contract  type  (high  risk  for  cost-type  contracts  and  low 
risk  for  fixed-price  contracts)  and  initial  percentage  of 
management  reserve  (MR)  (high  risk  for  MR  >  7.5%  and  low 
risk  for  MR  <  7.5%),  respectively.  If  both  the  contractor 
and  the  government  perceptions  of  risk  were  in  alignment 
with  the  actual  risk  involved  in  a  contract,  then  the 
expectation  was  to  experience  minimal  consequence.  If, 
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however,  either  or  both  parties  failed  to  accurately  assess 
the  actual  risk,  then  the  expectation  was  consequences,  such 
as  cost  overrun/underrun.  Consequently,  it  was  argued  that 
cost  overrun/underrun  is,  in  part,  a  function  of  the 
alignment  between  the  actual  risk  and  each  parties 
perceptions  of  the  risk.  In  order  to  investigate  the 
relationship  between  risk  alignment  and  cost  performance, 
two  hypotheses  were  developed  and  presented  in  Chapter  II. 

Results  of  Hypothesis  One 

Hypothesis  one  stated  cost  underruns  would  be  highest 
if  both  the  government  and  the  contractor  perceptions  of 
risk  were  higher  than  the  actual  risk.  In  this  scenario,  it 
was  believed  that  the  overestimation  of  risk  would  lead  to 
an  under-utilization  of  funds,  resulting  in  an  overall  cost 
underrun.  This  scenario  was  indicated  by  the  group  of 
contracts  in  risk  alignment  combination  number  eight,  where 
an  actual  risk  level  of  low  was  accompanied  by  government 
and  contractor  high  risk  perceptions. 

Based  on  the  variables  selected  and  the  analytical 
model  used,  hypothesis  one  was  not  supported.  Results 
showed  that  the  mean  value  of  final  cost  overrun/underrun 
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(FCO/U)  for  combination  number  eight  was  0.9866.  This 
indicates  that,  on  the  average,  contracts  in  this  group 
exhibited  an  underrun,  as  expected,  of  approximately  1.14%. 
The  mean  value  for  combination  number  eight,  however,  was 
not  significantly  different  from  the  other  combinations  when 
using  the  ANOVA  procedure.  This  may  have  been  due  in  part 
to  the  small  sample  size  for  this  combination  (3  cases) . 
Results  of  the  t-test  conducted,  however,  did  indicate  a 
statistically  significant  difference  between  combination 
number  eight  and  combination  number  one — represented  by  an 
underestimation  of  risk  by  both  the  government  and  the 
contractor.  While  not  conclusive  in  the  support  of  the 
underrun  hypothesis,  this  does  partially  support  the  belief 
that  an  overestimation  of  risk  by  both  parties  will  result 
in  lower  levels  of  cost  growth. 

Results  of  Hypothesis  Two 

Hypothesis  two  stated  cost  overruns  would  be  highest  if 
both  the  government  and  the  contractors  perceptions  of  risk 
were  lower  than  the  actual  risk.  In  instances  where  risk 
was  underestimated,  the  expectation  was  that  the  contract 
would  experience  a  cost  overrun,  because  the  parties  had 
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failed  to  adequately  provide  "safeguards"  to  help  minimize 
the  consequences  of  the  actual  risk  level.  This  scenario 
was  indicated  by  the  group  of  contracts  in  risk  alignment 
combination  number  one,  where  an  actual  risk  level  of  high 
was  accompanied  by  government  and  contractor  low  risk 
perceptions . 

Based  on  the  variables  selected  and  the  analytical 
model  used,  hypothesis  two  was  partially  supported,  however 
the  results  were  inconclusive.  Results  showed  the  mean 
value  of  final  cost  overrun/underrun  (FCO/U)  for  contracts 
in  combination  one  was  1.2264.  This  indicates  that,  on  the 
average,  contracts  in  this  group  exhibited  overruns,  as 
expected,  of  approximately  22.6%.  Additionally,  the  ANOVA 
procedure  indicated  the  mean  value  of  FCO/U  for  combination 
one  was  statistically  different  from  five  of  the  other  risk 
alignment  combinations.  The  significance  of  these  findings, 
however,  is  limited  by  the  fact  that  the  assumption  of  equal 
variances  within  the  combinations  was  violated. 

Because  of  the  ANOVA  limitation  on  equal  variances,  a 
two-sample  t-test  was  conducted  as  an  alternative  analysis 
in  the  hope  of  validating  the  ANOVA  results.  Results  from 
this  test  indicate  that  the  mean  value  of  FCO/U  for 
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combination  one  is,  in  fact,  significantly  different  at  the 
.05  level  from  all  but  one  risk  alignment  combination.  The 
exception  was  the  difference  between  combination  one  and 
combination  number  four,  where  both  parties  perceptions  of 
high  risk  were  in  line  with  an  actual  high  risk 
categorization.  While  not  conclusive,  this  result  partially 
confirms  the  hypothesis  that  an  underestimation  of  risk  by 
both  parties  will  create  a  higher  likelihood  of  experiencing 
a  contract  cost  overrun. 

IMPLICATIONS 

The  elements  of  risk — uncertainty,  consequence, 
safeguard — are  constants  in  DoD  acquisition.  The  theory 
behind  risk  management  discussed  in  this  study  suggests  a 
proactive  stance  towards  developing  safeguards  helps  to 
minimize  the  consequences  inherent  in  decisions  involving 
uncertainty.  One  of  the  keys  to  proactive  risk  management 
involves  an  accurate  assessment  of  the  actual  risks 
involved.  Failure  to  accurately  assess  the  risk  leads  to 
consequences  which,  most  often,  are  damaging  in  nature — as 
shown  by  the  Sergeant  York  and  AMRAAM  examples  presented  in 
Chapter  II. 
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An  extension  of  an  accurate  risk  assessment  is  an 


alignment  of  the  actual  risk  in  a  contract  with  the  level  of 
risk  perceived  by  both  the  government  and  the  contractor. 

A  misalignment  of  risk  could  result  in  damaging 
consequences,  such  as  poor  cost  performance.  As  cost 
performance  becomes  an  increasingly  growing  concern  in  DoD 
acquisition,  the  need  to  understand  the  reasons  for  poor 
cost  performance  also  increase.  The  intent  of  this  study 
was  to  determine  if  the  alignment  of  perceived  risk  to 
actual  risk  provided  some  insight  into  cost  performance 
behavior . 

The  statistical  results  do  not  completely  confirm  the 
hypotheses  presented  in  this  study.  They  do,  however, 
strongly  suggest  a  relationship  exists  between  the  alignment 
of  risk  and  the  cost  performance  experienced  on  a  contract. 
The  implications  for  DoD  contract  managers  is  that  proper 
risk  management  is  a  critical  element  in  the  effort  to 
achieve  cost  targets.  One  part  of  proper  risk  management  is 
the  ability  to  accurately  analyze  the  actual  risk  in  a 
contract  and  then  take  the  necessary  steps  to  develop 
"safeguards"  capable  of  mitigating  the  damaging  consequences 
of  that  inherent  risk.  Failure  to  adequately  align  the 
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perceptions  of  risk  with  the  actual  risk  has  been  shown  by 
this  study  to  create  a  tendency  towards  poor  cost 
performance.  Most  significantly,  an  underestimation  of  risk 
by  both  parties  could  result  in  the  highest  degree  of  cost 
growth. 

LIMITATIONS 

The  major  limitation  in  this  study  was  in  the 
development  of  the  independent  variables.  The  proxies  used 
to  operationalize  the  perceptions  of  risk  and  actual  risk 
were  theoretical  in  nature  and,  as  such,  their  effectiveness 
had  not  been  tested. 

One  of  the  indications  of  less-than-ideal  proxy 
selection  was  in  the  reported  correlation  between  contract 
type  and  phase  of  procurement.  The  strong  correlation  of 
.556,  however,  was  not  totally  unexpected.  In  fact,  the 
standard  practice  within  DoD  procurement  is  to  favor  cost- 
type  contracts  during  development  and  fixed-price  contracts 
during  production.  The  utilization  of  contract-type  in  this 
manner  is  accomplished  in  part  because  of  the  government's 
understanding  of  the  differences  in  risk  between  R&D  and 
production  phases.  While  the  high  correlation  tends  to 
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weaken  the  strength  of  the  findings,  it  does  not  totally 
invalidate  the  results,  as  would  be  the  case  with  a  perfect 
correlation  of  1.0. 

Additionally,  while  the  ANOVA  procedure  is  ideally 
suited  for  examining  the  variances  in  mean  values,  its  use 
may  not  have  been  totally  appropriate  for  this  data  set  due 
to  the  assumptions  it  imposes.  In  order  to  verify  the  ANOVA 
results,  a  t-test  had  to  be  conducted.  The  t-test  partially 
confirmed  the  validity  of  the  ANOVA  results,  but  it  also 
introduced  the  possibility  of  family-wise  error  associated 
with  multiple  t-test  comparisons.  This,  again,  tends  to 
weaken  the  value  of  the  findings. 


RECOMMENDATIONS  FOR  FUTURE  RESEARCH 

This  study  was  an  introductory  approach  into  the 
investigation  of  risk  alignment  and  its  effect  on  cost 
performance.  That  being  the  case,  there  are  many  areas  for 
future  improvement. 

The  most  notable  recommendation  is  the  development  and 

t 

usage  of  more  accurate  proxies  for  the  variables  of 
interest.  While  the  theory  behind  the  need  for  proper  risk 
alignment  has  been  established,  the  variables  used  to  test 
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that  theory  were  very  tentative  and  untested.  The 

development  of  more  suitable  proxies  for  actual  risk  and 

government  and  contractor  perceptions  of  risk  may  produce  i 

more  conclusive  results.  Specifically,  proxies  which  are  ^ 

independent  from  one  another  would  eliminate  the  high 

correlations  experienced  with  contract  type  and  phase  of 

procurement . 

Another  area  of  improvement  would  be  to  obtain  a  larger 
sample  of  some  of  the  key  risk  alignment  combinations.  One 
of  the  limiting  factors  in  statistically  proving  the 
difference  in  mean  FCO/U  values  was  the  small  number  of 
contracts  in  the  group  representing  overestimation  of  risk 
by  both  parties  (combination  number  eight) . 

The  final  recommendation  would  be  to  investigate  the 
theoretical  basis  for  the  behavior  of  the  risk  alignment 
combinations  as  a  continuum.  While  this  study  focused  on 
the  behavior  of  the  extreme  scenarios  (combinations  one  and 
eight) ,  there  may  exist  the  possibility  of  confirming  a 
sequential  relationship  among  the  eight  combinations.  A 

1 

greater  understanding  of  the  consequences  associated  with 
each  of  the  possible  risk  alignment  combinations  may  provide 
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DoD  acquisition  managers  with  more  information  to  use  in 
reducing  poor  cost  performance. 

) 

A.  ^ 
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